CULTURAL  AWARENESS  AND  NEGOTIATION  SKILLS  TRAINING: 
EVALUATION  OF  A  PROTOTYPE  SEMI-IMMERSIVE  SYSTEM 


Paula  J.  Durlach* 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Orlando,  FL  32826 

Timothy  G.  Wansbury 

U.S.  Army  Research,  Development,  and  Engineering  Command, 
Simulation  and  Training  Technology  Center 
Orlando,  FL  32826 

Jeffery  G.  Wilkinson 
MYMIC,  LLC 
Portsmouth,  VA  23704 


ABSTRACT 

The  effectiveness  of  BiLAT,  a  game-based  prototype 
for  training  negotiation  skills  in  an  Iraqi  cultural  setting, 
was  assessed.  The  prototype  aims  to  train  deliberate 
preparation  behaviors,  trust-  and  relationship-building 
strategies,  and  familiarity  with  relevant  cultural 
expectations  and  norms.  Training  is  supported  by 
automated  coaching  (during  simulated  negotiation 
meetings)  and  interactive  post-meeting  recaps. 
Effectiveness  was  measured  via  scores  on  an  independent 
situation  judgment  test,  completed  by  Soldiers  both  before 
and  after  training.  Each  time  administered,  the  test  yields  a 
single  score  indicating  the  degree  to  which  an  individual’s 
answers  concur  with  experts’  answers.  The  scores  of 
Soldiers  without  prior  negotiation  experience  increased 
significantly  from  pre-training  to  post-training.  The  scores 
of  Soldiers  with  prior  negotiation  experience  failed  to 
increase;  however,  their  initial  pre-training  scores  were 
already  relatively  high.  The  results  suggest  that  a  relatively 
short  amount  of  training  with  BiLAT  (less  than  three 
hours)  was  sufficient  to  increase  the  knowledge  level  of 
novice  negotiators.  It  remains  to  be  determined  whether 
only  novices  can  benefit  from  training  with  BiLAT,  or 
whether  more  experienced  Soldiers  might  also  gain  some 
benefit,  with  greater  amounts  of  training.  In  general. 
Soldier  response  to  the  training  experience  was  very 
positive,  despite  the  fact  that  more  than  50%  of  them  said 
they  played  games  rarely  or  never.  These  results  are 
significant  because  they  provide  new  evidence  that  game- 
based  tools  such  as  the  BiLAT  can  in  fact  be  used  within 
certain  conditions  to  effectively  train  Soldiers. 

1.  INTRODUCTION 


guided  by  information  gained  during  preparation  along 
with  information  revealed  by  a  negotiating  counterpart 
during  the  meeting  (Fisher  &  Ury,  1991;  Wunderle, 
2007).  Information  provided  by  the  counterpart  can  be 
explicit  and  clear  or  it  may  be  subtle  and  difficult  to 
decipher.  A  negotiator  needs  to  interpret  the 
counterpart’s  motivations  and  positions  based  not  only 
on  background  research,  but  also  on  how  the  counterpart 
reacts  during  the  meeting.  That  interpretation  can  be 
especially  difficult  if  the  negotiating  partner  is  from  a 
different  culture  with  unfamiliar  social  norms  and 
customs.  U.S.  Soldiers  are  increasingly  being  required  to 
conduct  formal  meetings  and  negotiations  with 
individuals  in  different  cultural  settings.  As  recent 
experiences  in  Iraq  and  Afghanistan  have  shown,  success 
or  failure  in  these  types  of  engagements  can  have 
profound  tactical  and  strategic  consequences.  Soldiers 
must  be  trained  in  these  skills  in  order  to  function 
optimally  in  these  new  operational  settings. 

The  US  Army  has  recognized  the  need  to  provide 
additional  training  and  to  better  prepare  Soldiers  for 
conducting  bi-lateral  engagements  and  negotiations. 
The  Army  has  taken  a  number  of  steps  to  integrate  this 
instruction  during  pre-deployment  training.  Limited  pre¬ 
deployment  training  occurs  at  home  station  using  live 
role  players  while  “graduation  exercises”  in  negotiation 
training  are  also  provided  at  the  Combat  Training 
Centers  (e.g..  National  Training  Center  (NTC)  and  Joint 
Readiness  Training  Center  (JRTC)).  Despite  the  efforts 
being  taken  to  address  this  new  training  requirement, 
some  experts  argue  that,  “While  this  training  is  an 
important  development,  it  is  not  sufficient.”  (Tressler, 
2007)  As  a  result,  additional  tools  and  methods  are 
required  to  improve  proficiency  and  to  better  prepare 
Soldiers  for  this  type  of  activity  in  the  future. 


Successful  negotiation  requires  good  preparation 
and  adherence  to  a  negotiation  strategy  (e.g.,  win-win). 
Strategy  application  requires  real-time  decision  making 
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U.S.  Army  Research,  Development,  and  Engineering 
Command,  Simulation  and  Training  Technology  Center; 
the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences;  and  the  U.S.  Army  Research 
Laboratory,  Human  Research  and  Engineering 
Directorate;  has  developed  a  game -based  training 
environment,  which  allows  Soldiers  to  develop  the 
requisite  skills  for  this  new  challenge.  The  system 
(Enhanced  Learning  Environments  with  Creative 
Technologies  for  Bi-Lateral  Negotiation  (BiLAT), 
provides  trainees  with  an  opportunity  to  develop  the 
necessary  skills  by  learning  how  to  prepare  for  and 
conduct  a  series  of  bilateral  negotiations  with  different 
simulated  characters  in  order  to  achieve  objectives 
related  to  support  and  stability  operations. 

The  BiLAT  prototype  contains  five  scenarios,  each 
situated  in  an  Iraqi  cultural  context.  The  scenarios  were 
built  to  support  the  accomplishment  of  specific  learning 
objectives  which  were  developed  in  conjunction  with  the 
U.S.  Army  Command  and  General  Staff  College,  School 
for  Command  Preparation  (SCP),  located  at  Ft. 
Leavenworth,  KS.  The  strategic  objectives  are  to  train 
deliberate  preparation  behaviors  and  trust-  and 
relationship-building  strategies,  which  enable  a 
negotiator  to  be  successful  under  difficult,  operational 
settings.  Another  aim  is  to  familiarize  the  trainee  with 
how  to  employ  knowledge  of  local  social  norms  to  make 
more  effective  judgments  about  appropriate  actions 
within  a  cross-cultural  negotiation.  A  more  detailed 
discussion  of  the  learning  objectives  and  the  way  they 
were  built  into  the  prototype  can  be  found  in  Kim,  Hill, 
Durlach,  Lane,  Forbell,  Core,  Marsella,  Pynadath,  and 
Hart  (submitted  for  publication). 

The  first  version  of  the  BiLAT  was  delivered  to  the 
SCP  in  October  2006  and  was  first  used  to  train  senior 
Army  officers  attending  the  Brigade  and  Battalion 
Commanders’  Pre-Command  Course  in  January  2007.  A 
number  of  usability  tests  were  conducted  over  the  next 
several  months  with  Soldiers  of  different  ranks  assigned 
to  units  at  Ft.  Leavenworth,  KS;  Ft.  Dram,  NY;  Ft. 
Campbell,  KY;  and  at  Ft.  Hood,  TX.  The  research  effort 
being  reported  upon  here  involved  an  experiment 
conducted  December  11-12,  2007,  with  31  Soldiers 
assigned  to  the  US  Army’s  10th  Mountain  Division  at  Ft. 
Dram,  NY. 

According  to  Hays  (2005),  there  is  an  increasing 
advocacy  for  the  use  of  games  for  military  instruction, 
despite  the  fact  that  their  training  effectiveness  is  poorly 
documented.  Proponents  of  training  games  argue  that 
younger  Soldiers  are  part  of  the  “digital”  generation,  and 
therefore,  will  respond  well  to  the  use  of  games  in 
training.  However,  other  important  research  has  shown 
that  the  majority  of  Soldiers  do  not  play  video  games  on 
a  frequent  basis.  The  research  also  suggests  that  this  fact 
may  have  an  impact  on  training  outcomes  (Belanich, 


Orvis,  Moore,  Horn,  and  Solberg,  2007).  Perhaps  too 
much  attention  has  been  given  to  the  media  rather  than 
the  message.  The  form  of  training  (e.g.,  classroom  vs. 
games)  is  likely  not  the  major  determinant  of 
effectiveness.  Rather,  that  is  determined  by  the 
instructional  design,  adherence  to  known  principles  of 
learning,  and  the  alignment  with  learning  objectives. 

The  current  trend  for  game-based  training  is  at  least 
partly  motivated  by  the  desire  to  expand  the  contexts  in 
which  training  can  occur  (anytime-anywhere),  although 
Hays  (2005)  suggests  that  games  may  be  best  used  in  a 
blended  environment  (i.e.,  as  an  adjunct  to  classroom 
instruction.)  In  any  case,  the  purpose  of  this  research 
effort  was  to  assess  whether  the  BiLAT  prototype,  a 
game -based  training  tool,  actually  met  its  objectives  of 
training  deliberate  preparation  behaviors,  trust-  and 
relationship-building  strategies,  and  familiarity  with 
relevant  cultural  expectations  and  norms. 

2.  OVERVIEW  OF  BiLAT 

Figure  1  depicts  a  schematized  version  of  the 
trainee’s  experience  in  BiLAT.  The  prototype  contains 
five  scenarios  reflecting  actual  negotiation  situations 
reported  to  have  been  given  by  Soldiers  with  combat 
experience  during  Operation  Iraqi  Freedom.  In  each 
scenario,  the  trainee  first  receives  an  overview  briefing 
for  the  scenario  and  a  goal(s)  to  accomplish  in  the 
training  exercise  (e.g.,  solve  the  problem  -  why  are  the 
Iraqi  merchants  not  using  the  new  town  market). 
Achieving  the  goal(s)  entails  reaching  specified 
agreements  or  obtaining  additional  key  information  from 
different  virtual  characters  in  the  exercise  (e.g.,  get 
police  cooperation  for  security  at  the  market). 

The  trainee  begins  each  exercise  by  conducting 
research  in  preparation  for  the  meeting  engagement  with 
the  first  virtual  character  in  the  scenario  (e.g.,  the  police 
chief).  The  principal  phases  for  each  meeting  cycle  are: 
preparation,  meeting/negotiation,  and  after  action  review 
(AAR).  The  preparation  phase  includes  a  number  of  sub¬ 
tasks  to  include  establishing  meeting  objectives, 
conducting  research,  requisitioning  necessary  materials, 
selecting  and  rehearsing  with  an  interpreter,  and 
finalizing  the  meeting  plan. 

Each  BiLAT  scenario  has  been  designed  so  that 
a  trainee  will  be  required  to  meet  with  a  minimum  of 
three  virtual  characters  in  order  to  successfully 
accomplish  all  of  the  objectives  for  an  exercise.  In  many 
cases,  the  trainee  will  have  to  meet  with  a  virtual 
character  on  more  than  one  occasion  in  order  to  continue 
with  the  scenario.  Once  the  specific  objectives  for  the 
meeting  are  accomplished,  the  trainee  is  rewarded  and 
granted  access  to  one  or  more  additional  characters  in 
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Fig  1.  Organization  of  BiLAT  Phases 


order  to  continue  the  exercise.  Each  meeting 
engagement  is  approached  in  the  same  manner  with  the 
process  (e.g.,  preparation,  meeting/negotiation,  AAR) 
repeating  itself  until  all  of  the  designated  objectives  for 
the  scenario  are  satisfied. 

During  the  preparation  phase  (Prep  in  Figure  1),  the 
trainee  is  situated  in  a  virtual  command  post  where  he 
conducts  the  required  research  and  gathers  information 
about  the  chosen  meeting  character  by  accessing  a 
number  of  different  resources.  Examples  include 
reviewing  reports  and  simulating  discussions  with  unit 
intelligence  officers,  civil  affairs  officers,  media  sources, 
as  well  as  members  of  the  local  population.  The  trainee 
must  analyze  the  raw  “intel  data”  and  decide  how  much 
of  the  information  is  important  in  preparation  for  the  up¬ 
coming  meeting. 

The  trainee  then  organizes  the  information  by 
completing  the  “Prep-Sheet,”  a  tool  which  was 
developed  previously  by  Subject  Matter  Experts  for 
negotiation  training  at  the  JRTC  at  Ft.  Polk,  LA  and 
adopted  for  use  in  this  prototype.  The  Prep-Sheet  is  an 
effective  tool  for  organizing  research  and  planning  one’s 
meeting  strategy.  The  information  accessed  during  the 
research  process  and  added  to  the  Prep-Sheet  provides 
the  basis  for  actions  the  trainee  will  be  able  to  take 
during  the  subsequent  meeting.  For  example,  if  the 
trainee  accesses  and  records  information  that  the  police 
chief  is  a  soccer  fan,  the  trainee  will  be  able  to  discuss 
soccer  with  the  police  chief  if  he  chooses  to  do  so  during 


the  meeting.  If  he  fails  to  uncover  and  record  this 
information  during  his  research,  this  option  will  not  be 
available  during  the  meeting. 

Following  completion  of  the  Prep-Sheet,  the  trainee 
can  simulate  requisitioning  items  which  might  be  of  use 
during  the  meeting  or  negotiation  session.  He  can  select 
a  virtual  interpreter  from  a  group  of  three  which  are 
provided  for  him  in  the  game  environment  and  he  can 
simulate  a  rehearsal  with  the  interpreter  before  beginning 
the  meeting  engagement.  The  trainee  can  access 
additional  information  relating  to  culture  awareness  and 
he  can  simulate  a  final  preparation  and  review  of  his 
meeting  plan  with  a  “trusted  virtual  agent”  (e.g.,  virtual 
battalion  executive  officer).  Once  the  trainee  feels  the 
preparation  phase  is  complete,  he  can  elect  to  go  on  to 
the  meeting  phase  of  the  exercise. 

The  meeting  is  situated  in  the  “world”  of  the 
meeting  counterpart  (e.g.,  the  police  chiefs  office). 
Figure  2  shows  a  screen  shot  from  a  BiLAT  meeting 
engagement.  Participants  choose  their  actions  from  a  list 
of  options  posted  on  the  left  side  of  the  screen.  Their 
choice  is  reiterated  in  white  text  in  the  dialog  box  on  the 
right.  The  virtual  character’s  response  can  be  heard 
orally  and  is  written  in  yellow  text  in  the  dialog  box  on 
the  right.  The  BiLAT  uses  advanced  intelligent  tutoring 
techniques  to  provide  a  coaching/tutoring  capability 
during  the  meeting  engagement.  Suggestions  from  the 
coach  appear  as  blue  text  in  the  dialog  box  on  the  right. 
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In  order  to  interact  with  the  simulated  counterpart, 
the  player  chooses  from  a  menu,  with  options  sorted  into 
one  of  four  categories  -  “Say”  (e.g.,  Greet  in  Arabic), 
“Ask”  (e.g.,  Do  you  know  why  people  are  not  using  the 
market?),  “Do”  (e.g.,  remove  sunglasses),  and  “Give” 
(e.g.,  offer  first  aid  kits).  Each  successful  meeting 
engagement  consists  of  an  opening  phase,  a  negotiation 
phase,  and  a  closing  phase.  Since  each  of  these  scenarios 
is  set  in  an  Iraqi  context,  it  is  important  for  the  trainee  to 
follow  local  Iraqi  social  customs  throughout  the  meeting 
engagement  (i.e.,  do  not  attempt  to  get  down  to  business 
until  sufficient  social  niceties  have  been  exchanged 
and/or  the  counterpart  brings  up  a  topic  related  to  the 
objective).  If  the  proper  steps  are  not  followed,  the 
virtual  character  will  react  negatively  (e.g.,  exhibit 
frustration  and  eventually,  anger).  If  the  trainee  persists 
in  not  following  the  appropriate  cultural  norms,  the 
meeting  partner  may  terminate  the  meeting. 

At  any  point  during  the  meeting,  the  trainee  can  ask 
the  virtual  character  to  negotiate.  In  turn  the  virtual 
character  can  agree  or  disagree,  based  in  large  part  upon 
the  degree  of  trust  that  has  been  established  between  the 
trainee  and  the  character  as  well  as  the  degree  of 
conformance  to  social  norms  during  the  meeting. 

In  the  negotiation  phase,  the  trainee  and  virtual 
character  make  offers  back  and  forth  until  an  agreement 
is  reached  or  until  the  trainee  decides  that  no  deal  can  be 
achieved  during  this  session.  The  underlying  negotiation 
principle  used  in  the  BiLAT  is  the  “Win-Win”  strategy 
(i.e.,  assumes  that  each  negotiation  partner  has 
something  to  offer  which  will  be  of  value  to  the  other 
partner  and  an  agreement  can  be  reached  where  both 
partners  benefit).  Automated  coaching  is  provided  to  the 
trainee  in  text  messages  throughout  the  negotiation 
phase.  The  models  that  control  character  responses  and 
provide  coaching  are  described  in  Kim,  et  al.  (submitted 
for  publication).  Once  the  negotiation  has  concluded,  the 
partners  enter  the  closing  phase  of  the  meeting  during 
which  more  interaction  and  social  niceties  are  continued. 
Many  times,  success  is  rewarded  during  this  part  of  a 
meeting  engagement  with  the  virtual  character  revealing 


additional  information  that  will  be  useful  in  subsequent 
meeting  engagements. 

The  final  phase  of  each  BiLAT  exercise  is  the  After- 
Action-Review  (AAR).  The  AAR  begins  with  a 
summary  assessment  of  how  the  player  performed  in  the 
meeting  followed  by  a  presentation  from  an  automated 
tutor  which  provides  detailed  feedback  on  the  meeting 
engagement.  The  AAR  includes  a  replay  of  the  meeting 
in  which  the  tutor  reviews  the  meeting  interactions  and 
asks  questions  (answered  from  a  menu)  which  are 
intended  to  promote  reflection  about  ways  the  learner 
might  improve  performance  in  the  future.  Following  the 
AAR,  the  exercise  continues  with  another  meeting  with 
the  same  character  or  with  research  beginning  in 
preparation  for  a  meeting  with  a  new  character.  The 
combination  of  the  summary  assessment  and  the  detailed 
feedback  from  the  automated  tutor  provides  an  effective 
assessment  of  the  trainee’s  performance  and  can  be  used 
by  the  trainee  to  improve  performance  during  follow-on 
meeting  engagements 


3.  THE  TRAINING  EVALUATION  SITUATION 
JUDGMENT  TEST 

In  order  to  evaluate  the  training  effectiveness  of 
BiLAT,  we  created  an  independent  assessment  of 
respondents’  ability  to  apply  negotiation  strategies  and 
tactics,  appropriate  for  an  Iraqi  cultural  context.  The 
assessment  used  a  variation  of  a  Situational  Judgment 
Test  (SJT).  SJTs  are  often  used  for  personnel  selection 
and  prediction  of  job  performance  (O’Connell,  Hartman, 
McDaniel,  Grubb,  &  Lawrence,  2007).  SJT  items 
typically  begin  with  a  brief  scenario  which  is  then 
followed  by  several  statements  representing  potential 
interpretations,  responses,  or  actions  to  the  SJT  scenario. 
A  respondent  must  judge  the  appropriateness  of  the 
statements  as  related  to  the  scenario.  The  statements  do 
not  necessarily  have  objective,  correct  or  incorrect 
answers;  rather  responses  indicate  a  pattern  of  judgment. 
To  score  the  test,  an  individual’s  pattern  is  compared  to 
normative  patterns  collected  from  groups  with  known 
characteristics  (e.g.,  experts  vs.  novices,  or  leaders  vs. 
followers).  The  respondent’s  “score”  is  determined  by 
the  degree  to  which  their  pattern  conforms  to  the 
normative  pattern  from  the  “desirable  group.” 

The  project  partners  collaborated  to  create  more  than 
20  initial  scenarios  with  associated  statements  in  the  first 
version  of  the  SJT.  This  pool  of  scenarios  was  evaluated 
and  whittled  down  to  a  smaller  set  in  consultation  with 
expert  instructors.  This  reduced  set  was  then  given  to 
four  independent  subject  matter  experts  (SMEs),  who 
judged  the  “goodness”  of  the  statements  associated  with 
each  scenario  (i.e.,  provided  their  answers).  A  subset  of 
these  scenarios  and  statements  were  then  selected, 
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choosing  ones  which  maximized  SME  agreement  and 
provided  coverage  for  BiLAT’s  top-level  learning 
objectives.  The  final  SJT  included  nine  scenarios  with 
three  or  four  evaluation  statements  per  scenario.  Pair 
wise  agreement  between  the  SMEs  (as  measured  by 
correlation)  on  the  final  subset  was  .92  or  better.  An 
example  of  one  of  the  SJT  items  is  given  in  Table  1. 


Soldiers  had  previously  been  deployed  at  least  once; 
most  of  these  had  been  to  Afghanistan  (20);  relatively 
few  had  been  to  Iraq  (six).  When  asked  how  often  they 
played  computer  or  video  games,  only  eight  participants 
(26%)  reported  playing  a  few  times  a  week  or  more. 
Seventeen  (55%)  said  they  never  or  rarely  played  video 
games. 


Table  1.  Sample  Item  from  Situation  Judgment  Test  4.2  Equipment  and  Materials 


MAJ  O’Rourke  is  about  to  meet  with  a  local  Iraqi  leader  for  the 
first  time.  The  MAJ  is  concerned  about  the  potential  outcome 
of  this  meeting  where  he  will  try  to  find  out  infonnation  about 
a  suspected  insurgent  group  in  the  area.  What  should  you  tell 
him? 

Not  to  worry  because 
whatever  happens  stays 
between  him  and  his 

Poor 

Moderate 

Good 

negotiation  partner.  If 
something  goes  wrong, 
it’s  not  a  big  deal. 

0  1 

2  3  4  5  6  7 

8  9  10 

It  will  help  if  he  has 
planned  for  the  possible 
effects  of  both  success 

Poor 

Moderate 

Good 

and  failure  of  the  meeting 
on  the  area  of  operations. 

0  1 

2  3  4  5  6  7 

8  9  10 

He  should  be  ready  to  put 
pressure  on  the  local  Iraqi 
leader  if  he  does  not 

Poor 

Moderate 

Good 

immediately  provide  the 
information  needed. 

0  1 

2  3  4  5  6  7 

8  9  10 

The  procedure  for  arriving  at  a  score  for  each 
individual  involved  first  standardizing  each  individual’s 
responses,  then  computing  the  correlation  between  the 
standardized  scores  and  the  average  of  the  standardized 
SME  scores.  This  produced  one  number,  ranging 
between  -1.0  and  1.0,  for  each  participant.  The  number 
represents  the  extent  to  which  a  participant’s  answers 
agreed  with  those  of  the  SMEs,  with  1.0  representing 
perfect  agreement  and  -1.0  representing  perfect 
disagreement. 

The  SJT  was  administered  twice  to  each  participant 
for  this  evaluation:  once  prior  to  training  with  BiLAT 
and  once  immediately  following  the  training  with 
BiLAT.  An  increase  in  the  SJT  score  from  pre-to-post 
training  would  suggest  that  a  participant’s  ability  to 
apply  negotiation  strategies  and  tactics,  appropriate  for 
Iraqi  culture,  had  benefited  as  a  result  of  the  training. 

4.  METHOD 

4.1  Participants 

Thirty-one  Soldiers  participated  in  the  study.  Of  the 
31,  15  were  enlisted  Soldiers  (E4  -  E7),  and  16  were 
commissioned  officers  (Ol  -  03).  All  but  five  of  the  31 


Eight  laptop  computers  were  used  for  the  training. 
Each  of  the  laptops  was  loaded  with  the  BiLAT  (version 
1.7)  software.  The  study  was  conducted  in  a  well-lit 
room  with  several  tables  and  chairs.  A  single  laptop  was 
placed  on  each  table.  Soldiers  were  asked  to  complete 
questionnaires  collecting  basic  background  and 
demographic  information  as  well  as  both  a  pre-study  and 
post-study  SJT. 

4.3  Procedure 

The  experiment  was  conducted  over  a  two-day 
period,  December  11-12,  2007.  Half  of  the  Soldiers  (15) 
participated  in  the  experiment  on  the  first  day  and  the 
remaining  Soldiers  (16)  participated  in  the  experiment  on 
the  second  day.  The  activities  for  each  day  were  divided 
into  three  sessions  —  a  morning  session  and  two 
afternoon  sessions.  Each  Soldier  participated  in  one 
morning  session  and  just  one  of  the  afternoon  sessions. 
The  morning  session  lasted  approximately  2.5  hours. 
Approximately  one  hour  was  devoted  to  preparation  for 
the  experiment  and  1.5  hours  were  devoted  to  the  initial 
BiLAT  training  exercise. 

Some  Soldiers  arrived  for  the  morning  session  with 
a  completed  demographics  questionnaire  and  pre-test 
SJT  which  they  had  received  from  the  Ft.  Drum  exercise 
coordinator  prior  to  the  study.  Those  who  did  not  bring 
these  documents  to  the  training  were  asked  to  complete 
them  both  immediately  upon  arrival.  Each  Soldier  was 
then  assigned  a  login  name,  a  partner,  and  a  time  to 
return  for  the  afternoon  session.  Partners  were  allowed  to 
work  together  during  the  morning  session;  but,  were 
assigned  different  afternoon  times. 

The  morning  sessions  began  once  the  paperwork 
was  completed.  One  member  of  the  study  control  group 
provided  an  overview  briefing  to  the  participants  which 
lasted  about  one  hour.  The  briefing  described  the  overall 
goals  of  the  research  project,  identified  the  strategic 
learning  objectives  for  the  BiLAT,  and  provided  a 
demonstration  of  the  BiLAT  itself.  This  “knobology” 
demonstration  showed  the  Soldiers  how  to  “play  the 
BiLAT  game.”  Since  the  group  of  test  subjects  included 
a  high  percentage  of  “non-gamers,”  the  Soldiers  were 
asked  to  initially  work  with  a  partner  while  using  the  first 
BiLAT  “Market”  scenario.  The  control  team  directed  this 
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approach  in  order  to  help  the  “non-gamers”  overcome 
any  problems  or  resistance  to  using  a  game-based 
training  tool  and  to  remove  this  as  an  obstacle  to  the 
learning  which  we  hoped  would  occur  during  the 
individual  training  sessions. 

The  study  team  circulated  the  room  and  answered 
questions  providing  help  or  advice  where  required.  The 
overall  goal  of  this  practice  session  was  to  ensure  that  all 
Soldiers  understood  the  mechanics  of  how  to  interact 
with  the  software.  After  this  practice  session,  Soldiers 
completed  a  short  questionnaire  providing  their  initial 
reactions  to  the  game.  The  Soldiers  participated  in  a 
brief  group  discussion  and  the  control  team  addressed 
any  outstanding  questions.  Soldiers  were  released  for 
lunch  and  were  reminded  about  what  time  to  return  in  the 
afternoon. 

The  afternoon  sessions  on  each  day  were  divided 
into  two  separate,  two-hour  training  periods.  Each 
Soldier  participated  in  just  one  of  the  afternoon  training 
periods.  Upon  arrival,  each  Soldier  chose  a  computer 
and  began  to  work  individually  on  a  new  BiLAT 
scenario  (Power  Scenario).  Initially,  the  study  team 
provided  help  only  if  a  Soldier  had  software  trouble  (e.g., 
software  seemed  to  “lock-up”);  however,  after 
approximately  60  minutes,  the  team  also  found  it 
necessary  to  provide  substantive  assistance  to  some 
Soldiers  who  asked  for  help. 

The  individual  training  sessions  lasted 
approximately  90  minutes.  At  the  end  of  the  90  minute 
period,  the  Soldiers  were  instructed  to  complete  their 
current  meeting  if  they  were  in  the  meeting  or  AAR 
phase  of  the  exercise.  They  were  instructed  not  to  start  a 
new  meeting.  If  the  trainees  were  in  the  research  or 
preparation  phase  for  another  meeting  engagement,  they 
were  simply  asked  to  stop  work.  When  all  of  the  Soldiers 
were  ready,  each  was  asked  to  complete  the  post-training 
SJT  and  a  usability  questionnaire.  Finally,  each  afternoon 
session  concluded  with  the  study  team  discussing  the 
training  exercise  with  the  Soldiers.  Their  feedback  was 
useful  as  we  look  forward  to  continuing  development  of 
the  BiLAT  prototype  in  the  future. 

Each  of  the  two  scenarios  used  (Market  in  the 
morning  session  and  Power  in  the  afternoon  sessions)  for 
the  training  had  three  potential  meeting  characters.  The 
trainees  were  required  to  obtain  certain  objectives  with 
one  character  before  they  could  move  on  and  meet  with 
the  next  character.  It  is  thought  by  those  familiar  with  the 
BiLAT  that  the  Market  scenario  is  an  easier  exercise  than 
the  Power  scenario;  however,  no  formal  assessment  of 
scenario  difficulty  has  been  conducted. 


5.  RESULTS 

5.1  Participant  Background 

Two  of  the  participants  said  they  were  familiar  with 
BiLAT,  but  had  never  used  it.  The  rest  said  they  had 
never  heard  of  the  BiLAT  before  preparation  for  the 
experiment.  Median  self-rated  knowledge  of  Arab  and 
Middle-Eastern  cultures  was  3,  on  a  1  to  7  scale,  where  1 
was  labeled  Minimal,  4  was  labeled  Competent,  and  7 
was  labeled  Expert.  Only  8  participants  rated  themselves 
as  Competent  or  better.  There  was  a  significant  positive 
association  between  self-rating  and  the  number  of 
months  spent  in  deployment  (Spearman  r  =  .43).  There 
was  also  a  significant  positive  association  between  self¬ 
rating  and  prior  negotiation  experience  (Spearman  r  = 
.54),  with  11  of  the  Soldiers  stating  they  had  prior 
negotiation  experience.  Ten  of  the  Soldiers  had  some 
previous  formal  training  in  negotiation  skills.  When 
asked  how  often  they  played  computer  or  video  games, 
only  eight  participants  reported  playing  a  few  times  a 
week  or  more.  Seventeen  said  they  never  or  rarely  played 
video  games. 

5.  2  Comparison  of  Pre-training  and  Post-training 
SJT  responses 

Pre-training  SJT  responses  were  significantly 
correlated  with  self-rated  knowledge  of  Middle  Eastern 
culture  (Spearman  r  =  .49)  and  post-training  SJT  scores 
were  significantly  correlated  with  previous  formal 
negotiation  training  (Spearman  r  =  .39).  To  take  these 
influences  into  account,  an  analysis  of  covariance  was 
used  to  assess  the  change  between  pre-  and  post-training 
SJT  scores.  SJT  scores  were  treated  as  a  repeated 
measure  and  were  analyzed  with  two,  two-level 
categorical  factors  (enlisted  vs.  officer  and  previous 
negotiation  experience),  and  two  covariates  (level  of 
previous  formal  training  and  self-rated  cultural 
knowledge).  However,  neither  of  the  covariates  actually 
accounted  for  a  significant  amount  of  variance  in  the 
analysis  of  covariance.  In  general,  SJT  scores  increased 
from  pre-training  to  post-training.  Mean  pre-training  SJT 
score  was  .62,  whereas  mean  post-training  SJT  score  was 
.76.  The  interpretation  of  this  increase  was  complicated 
by  two  significant  interactions.  One  was  between  time 
of  testing  (pre-  vs.  post)  and  enlisted  vs.  officer,  F(  1,  25) 
=  4.60,  p  <  .05,  and  the  other  was  between  time  of  testing 
and  previous  bilateral  negotiation  experienced,  F(  1,  25) 
=  5.51,/?  <  .05.  These  data  are  illustrated  in  Figure  3. 

As  can  be  seen  in  Figure  3,  the  post-training  SJT 
scores  depicted  were  fairly  equivalent  and  failed  to  differ 
significantly  among  one  another.  The  interactions 
mentioned  above  were  primarily  due  to  differences  in 
pre-training  SJT  scores.  The  pre-training  scores  were 
higher,  on  average,  for  officers  as  compared  to  enlisted 
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personnel  (solid  vs.  open  symbols).  They  were  also 
higher,  on  average,  for  Soldiers  who  had  prior  bilateral 
negotiation  experience  in  another  culture  (left  panel). 
Considering  only  participants  without  prior  negotiation 
experience  (right  panel),  the  increase  in  SJT  scores  from 
pre-  to  post-training  was  significant,  regardless  of  officer 
or  enlisted  status,  i(20)  =  4.39,  p  <  .01.  For  those  with 
prior  negotiation  experience,  there  failed  to  be  a 
statistically  significant  change. 


character.  We  analyzed  the  final  trust  score  for  each 
Soldier’s  first  meeting  with  Ali  (mean  10.3)  against  the 
trust  score  for  their  last  meeting  with  Ali  (mean  12.8). 
The  increase  from  first  to  the  last  trust  score  was 
significant,  7^(1,  20)  =  5.13,  p  <  .05,  suggesting  Soldiers 
were  becoming  more  adept  at  building  trust  over  time. 
In  addition,  a  person’s  best  end-of-meeting  trust  score 
was  significantly  correlated  with  total  number  of 
objectives  achieved  (Spearman  r  =  .49).  People  with 
higher  best  trust  scores  obtained  more  of  the  objectives. 


Pre-Training  Post-Training  Pre-Training  Post-Training 

Prior  Negotiation  Experience  No  Negotiation  Experience 


Fig  3.  Mean  pre-  and  post-training  SJT  scores,  shown 
separately  for  officer  and  enlisted  (solid  vs.  open 
symbols),  and  for  Soldiers  with  vs.  without  prior 
negotiation  experience  (left  vs.  right  panels).  Bars  denote 
95%  confidence  intervals. 


5.3  Performance  during  the  Power  Scenario  and 
Soldier  Feedback  on  BiLAT 

A  trainee  had  to  meet  and  negotiate  with  three 
separate  characters  (Ali,  Ismaa’el,  and  Hassan)  in  order 
to  successfully  complete  the  entire  Power  scenario. 
Players  repeatedly  met  with  Ali  until  they  achieved  the 
two  required  negotiation  agreements.  Once  these 
objectives  are  achieved,  access  to  the  next  character, 
Ismaa’el  was  granted,  and  so  on.  Fifteen  of  the  31 
participants  obtained  the  first  two  objectives  with  Ali; 
only  one  obtained  the  next  pair  of  objectives  with 
Isma’ael.  The  median  number  of  meetings  completed 
was  four  (range:  one  to  eight).  On  average,  participants 
spent  about  17.5  minutes  on  each  meeting. 

During  each  meeting  the  BiLAT  system  displays  a 
trust  meter  which  illustrates  how  well  the  virtual 
character  trusts  the  player.  Player  mistakes  tend  to 
decrease  the  amount  of  trust  exhibited  by  the  virtual 
character,  whereas  culturally  appropriate  actions  tend  to 
increase  trust.  The  trust  meter  scale  ranges  from  a  -5  to 
+15.  The  higher  the  value  on  the  trust  meter,  the  greater 
the  level  of  trust  between  the  trainee  and  the  virtual 


Post-training  Soldier  opinions  of  the  BiLAT 
prototype  and  the  training  experience  was  generally 
positive.  The  median  response  to  each  of  the  six  post¬ 
exercise  questions  about  their  training  experience  was  6. 
The  scale  for  possible  responses  ranged  from  1  (negative 
opinion)  to  7  (positive  opinion).  The  distribution  of 
responses  to  the  post-exercise  questions  is  shown  in 
Table  2. 

Table  2.  Responses  to  post-training  questions  (Scale  1-7) 


Number  Choosing  Alternatives 


Alternatives  1  (most 
negative)  to  7  (most 
positive) 

1 

2 

3 

4 

5 

6 

7 

Useful  training  or 
practice  in  bilateral 
negotiation 

0 

0 

0 

2 

4 

27 

5 

Useful  training  on 
key  Iraqi  cultural 
characteristics 

0 

0 

1 

4 

4 

16 

6 

Ease  of  using 
software 

9 

1 

1 

6 

6 

7 

10 

Ease  of  use  to  gather 
meeting  preparation 
information 

0 

0 

0 

2 

11 

6 

12 

Ease  of  use  to 
conduct  meeting 
phases 

0 

1 

3 

3 

8 

10 

6 

Helpfulness  of  AAR 

0 

2 

0 

8 

5 

6 

10 

CONCLUSIONS 

The  present  results  are  important  because  they 
provide  new  evidence  that  a  game-based  training  tool 
such  as  BiLAT  can  be  used  effectively  to  train  Soldiers. 
The  results  suggest  that  a  relatively  short  amount  of 
training  with  BiLAT  (about  three  hours)  was  effective  in 
increasing  the  knowledge  level  of  novice  negotiators,  as 
measured  by  the  SJT.  The  face  validity  of  the  SJT  as  a 
measure  of  relevant  knowledge  was  bolstered  by  its 
ability  to  discriminate  experienced  vs.  novice 
negotiators.  That  is,  the  finding  that  prior  to  training,  SJT 
scores  were  significantly  higher  for  experienced  as 
compared  to  novice  negotiators  suggests  that  the  SJT  is 
measuring  at  least  some  aspects  of  negotiation  skill. 
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It  remains  to  be  determined  whether  the  training 
effectiveness  of  BiLAT  is  limited  to  novices  only,  or 
whether  more  experienced  Soldiers  might  also  gain  some 
benefit.  The  exercise  control  team  had  access  to  each 
participant  for  only  a  single  day.  This  limited  the  amount 
of  time  that  Soldiers  were  available  to  interact  with 
BiLAT.  More  interaction  time  may  have  resulted  in  a 
measurable  benefit  to  experienced  as  well  as  novice 
Soldiers.  It  is  also  possible  that  more  experienced 
Soldiers  reaped  a  training  benefit,  but  that  the  SJT  was 
incapable  of  detecting  it.  We  have  not  analyzed  the 
psychometric  properties  of  the  SJT,  and  it  is  possible  that 
it  is  not  equally  sensitive  at  different  parts  of  its  scale.  It 
was  also  impossible  to  determine  what  impact  if  any  the 
initial  briefing  given  during  the  “knobology”  instruction 
had  on  the  test  results. 

Further  research  is  required  before  a  sound 
prescription  on  how  to  best  implement  training  with 
BiLAT  can  be  given.  This  study  alone  is  insufficient  to 
indicate  the  appropriate  training  audience  or  the 
appropriate  training  standard  (e.g.,  training  time  or  other 
performance  criterion).  Research  suggests  that  tailoring 
the  difficulty  level  of  the  scenarios  to  the  initial  skill 
level  of  the  trainee  would  be  an  optimum  approach 
(Bjork,  1994),  though  it  would  necessitate  assessing  that 
initial  skill  level. 

Access  to  Soldiers  for  only  one  day  also  limited  our 
ability  to  assess  retention.  As  shown  in  several  studies 
(Druckman,  &  Bjork,  1994),  performance  measures 
taken  immediately  after  training  do  not  necessarily 
coincide  with  those  taken  at  a  later  time.  Delayed 
measures  indicate  the  persistence  of  acquired  knowledge. 
Compared  to  immediate  assessment,  therefore,  delayed 
assessment  probably  provides  a  better  prediction 
regarding  the  transfer  of  training  to  real-life  situations. 
Note  that  pre-training  SJT  measures  correlated 
significantly  with  prior  negotiation  experience,  but  failed 
to  correlate  significantly  with  prior  formal  negotiation 
training.  Yet,  post-training  SJT  scores  did  show  a 
significant  positive  association  with  prior  formal 
training.  A  possible  interpretation  is  that  knowledge 
gained  by  prior  formal  negotiation  training  was 
cognitively  inaccessible  to  Soldiers  before  BiLAT 
training;  however,  BiLAT  training  acted  to  rekindle  this 
knowledge  and  make  it  accessible  after  training. 
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